Repository navigation
fix: hold the reload lock in short slices and stop on an unresponsive client - #450
Merged
Merged
Conversation
… client The reload pass held the validator operation lock for its whole run, including one fee recipient POST per key. On large nodes a pass can exceed the 30s wait budget of user operations, which then failed as busy (HTTP 503 on the Launchpad API). Key membership (steps 0-5) is still reconciled under one lock. Fee recipients and builder configs are then applied in slices that hold the lock for at most 1s and re-read the database inside the lock, so they never re-send a value a user has since replaced and skip keys deleted mid-pass. Reload fee recipient writes now time out after 10s. A request with no HTTP response (timeout, refused or reset) stops the pass; the next pass retries every key. HTTP errors concern one key and the pass continues, as before.
…nding When the fee recipient step stopped because the validator client gave no response, reload still applied every builder configuration, each waiting up to 10s. With the builder feature enabled this could delay the next pass for hours on large nodes. The slice helper now reports whether a step stopped the run. Reload skips the builder phase when the fee step stopped, and the builder step itself stops at the first request without a response. Builder errors keep a cause with only the HTTP status, never the response body, so echoed authorization data stays out of logs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The reload cron holds the validator operation lock for its whole pass, including one fee recipient POST per key, sent one after another. UI and Launchpad API operations wait at most 30s (
CLIENT_OPERATION_MAX_WAIT_MS) for the lock and then fail as busy (HTTP 503 for Lido CSM and Stader). 0.1.33 had no lock, so this is new in 0.2.0.On a node with many validators, or a client that is slow per write (for example, Lighthouse rewrites its validator definitions file on every POST), a pass can exceed 30s, and users get intermittent busy errors. If the validator client accepts connections but never answers, each POST waits the 120s default timeout, so the lock can be held for hours and blocks deletes.
Change
forEachInLockedSlices). Each slice re-reads the DB inside the lock, so it never re-sends a fee recipient a user has since replaced, and it skips keys deleted mid-pass. User operations now wait at most about one slice plus one request.ErrorOptions), andreceivedHttpResponse()checks it.setFeeRecipienttakes an optionaltimeout; other callers are unchanged.Tests
New
reloadLockSlices.unit.test.ts:Lint, build and all suites pass (brain unit 116 + 1 skipped, e2e 10, UI 65, common 13). The timing test passed 5 runs in a row.
Trade-offs